Synchronizing media in multiple devices

ABSTRACT

A system includes a processor and a memory. The memory stores instructions executable by the processor to receive first and second media units with respective first and second time stamps that are assigned based on a first clock cycle time and a data transmission rate, and to assign an adjusted time stamp to the second media unit based on the first clock cycle time, a second clock cycle time, the first time stamp, and the data transmission rate.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of U.S. patent applicationSer. No. 17/037,353, filed Sep. 29, 2020, entitled “SYNCHRONIZING MEDIAIN MULTIPLE DEVICES,” which is a continuation of U.S. patent applicationSer. No. 15/871,904, filed on Jan. 15, 2018, entitled “SYNCHRONIZINGMEDIA IN MULTIPLE DEVICES,” which claims priority to and all thebenefits of Indian Provisional Patent Application No. 201741008426,filed on Mar. 10, 2017, entitled “SYNCHRONIZING MEDIA IN MULTIPLEDEVICES,” all of which are incorporated herein by reference in theirentirety.

BACKGROUND

The use of streamed media data such as video data has increased rapidly.Videos are used for instructional purposes, to record and view sportsevents, to share personal and family events, to sell real estate, etc.Typically, a transmitter streams media data to a receiver periodicallybased on a transmission rate, e.g., 25 frames per second (fps).Nevertheless, while the receiver reproduces and/or records the receivedstreamed media, a difference between the timing of media unitsdetermined by the transmitter versus a timing that the receiver expectsmay cause difficulties at the receiver.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an exemplary media processing system forcapturing and mixing multiple media streams into an output media stream.

FIG. 2 is a graph showing transmitted media units and adjusting of mediaunits time stamps.

FIG. 3 is a graph showing a media unit discarded after being receivedoutside an expected time interval.

FIG. 4A is a graph showing a media unit accepted after being receivedoutside an expected time interval.

FIG. 4B is a graph showing a last media unit repeated until a next mediaunit is received.

FIG. 5 is an example flowchart showing a process for synchronization ofreceived media units.

FIG. 6 is another example flowchart showing a process forsynchronization of received media units.

DETAILED DESCRIPTION Introduction

Disclosed herein is a system including a processor and a memory. Thememory stores instructions executable by the processor to receive firstand second media units with respective first and second time stamps thatare assigned based on a first clock cycle time and a data transmissionrate, and to assign an adjusted time stamp to the second media unitbased on the first clock cycle time, a second clock cycle time, thefirst time stamp, and the data transmission rate.

The adjusted time stamp may be a nearest time stamp to the second timestamp based on the transmission rate, and is divisible by the secondclock cycle time.

The second clock cycle time may be less than the first clock cycle time.

The memory may store instructions executable by the processor to assignthe adjusted time stamp to the second media unit only upon determiningthat the second time stamp is within an expected time interval.

The instructions may include further instructions to discard the secondmedia unit upon determining that the second time stamp is outside theexpected time interval.

The instructions may include further instructions to discard the secondmedia unit only upon determining that the second time stamp is outsidethe expected time interval and within a second time threshold greaterthan the expected time interval.

The instructions may include further instructions to re-use the firstmedia unit instead of the second media unit upon discarding the secondmedia unit.

The instructions include further instructions to re-use the first mediaunit by assigning the adjusted time stamp to the first media unit andprocessing the first media unit with the adjusted time stamp as areplacement for the discarded second media unit.

The instructions may include further instructions to determine theexpected time interval based on the first clock cycle time and aplurality of time stamps associated with a plurality of last receivedmedia units.

The instructions may include further instructions to determine theexpected time interval based on an expected time stamp and a maximumdeviation threshold.

The instructions may include further instructions to prevent anadjustment of the second time stamp upon determining that the secondtime stamp is outside an expected time interval.

The instructions may include further instructions to determine a secondexpected time interval for a third media unit that is expectedsubsequent to the second media unit, based at least in part on thesecond time stamp.

The instructions may include further instructions to receive the thirdmedia unit, subsequent to the second media unit, with a third timestamp, and to assign an adjusted third time stamp to the third mediaunit based at least in part on the second time stamp.

The instructions may include further instructions to repeat the firstmedia unit upon determining that the second time stamp is outside anexpected time interval and outside a second time threshold greater thanthe expected time interval.

Further disclosed herein is a method including receiving first andsecond media units with respective first and second time stamps that areassigned based on a first clock cycle time and a data transmission rate,and assigning an adjusted time stamp to the second media unit based onthe first clock cycle time, a second clock cycle time, the first timestamp, and the data transmission rate.

The method may further include discarding the second media unit onlyupon determining that the second time stamp is outside the expected timeinterval and within a second time threshold greater than the expectedtime interval.

The method may further include repeating the first media unit upondetermining that the second time stamp is outside an expected timeinterval and outside a second time threshold greater than the expectedtime interval.

Further disclosed herein is a system including a client device computerprogrammed to transmit first and second media units. The system includesa media studio computer programmed to receive, from a media device,first and second media units with respective first and second timestamps that are assigned based on a first clock cycle time and a datatransmission rate, and to assign an adjusted time stamp to the secondmedia unit based on the first clock cycle time, a second clock cycletime, the first time stamp, and the data transmission rate.

The media studio computer may be further programmed to repeat the firstmedia unit upon determining that the second time stamp is outside anexpected time interval and outside a second time threshold greater thanthe expected time interval.

The media studio computer may be further programmed to discard thesecond media unit only upon determining that the second time stamp isoutside the expected time interval and within a second time thresholdgreater than the expected time interval.

Further disclosed herein is a computer that is programmed to execute anyof above method steps.

Yet further disclosed herein is a computer-readable medium, storingprogram instructions for executing any of above method steps.

EXEMPLARY SYSTEM ELEMENTS

FIG. 1 shows a block diagram of an exemplary media processing system 100for capturing and mixing multiple media streams into one or more outputmedia streams. As used herein, a media “unit” is a smallest set of datathat can be provided in a given digital media format, e.g., a singleframe in MPEG and other video formats is a media unit. Further as usedherein, a “stream” means a plurality of units of digital media data in apredefined format (e.g., MPEG, MP3, etc.) being provided to (input) orfrom (output) a device.

The media processing system 100 may include a media studio 105, one ormore media devices 115, and a display device such as a monitor 150. Theone or more media devices 115 may include an operator console 120, e.g.,a tablet computer, and may further include one or more cameras 125, oneor more viewers 130 and one or more media storage devices 135. The mediadevices 115 may be remote or local to the media studio 105 and may becoupled to the media studio 105 via at least one of a network 140 and adirect wired or wireless connection. A server 145 may also becommunicatively coupled to the media studio 105 and the media devices115 via the networks 140.

The media studio 105 may include a processor 110 that is programmed toreceive multiple media input streams, and to combine them into amultiview media item for viewing via the operator console 120 and/or themonitor 150. The media input streams are sometimes referred to herein asindividual media items. The multiview media item is an aggregated mediacontent item that includes one or more of the media input items (orstreams), arranged to be displayed substantially simultaneously on auser display, such as a touch screen. The media input items may bearranged such that they appear side-by-side, in rows, in apicture-in-picture format, etc. within the user display. In addition,the multiview media item may include, for example, a graphical userinterface which is displayed, e.g., on the operator console 120, andaccepts user inputs. As described below, the media studio 105 maycompress the multiview media item prior to providing it to one of themedia devices 115 (e.g., data provided according to MPEG or the like iscompressed) such as the operator console 120 and/or the storage devices135.

The media input streams may be visual and/or audio data such as videoscaptured by a video camera 125, sounds captured by a microphone, scenesfrom a security camera 125, etc. The media input streams may be, e.g.,streamed data or static data such as single digital photographs. Themedia studio 105 is further operable to receive commands from theoperator console 120, and to generate a media output item or streamaccording to the received commands. The media output item may includedata from one or more of the media input items.

As an example, the media studio 105 may receive four media input items.The media studio 105 may generate a multiview media item including eachof the four media input items, and transmit the multiview media item tothe operator console 120. A user of the operator console 120 may select,via a user interface, one of the four views in the multiview media itemto be included in the media output item. The media studio 105 processor110 may be programmed to generate the media output item that includesthe selected media input item, based on a command received from theoperator console 120. The media output item may be transmitted via anetwork 140 to be shared with viewers 130, outputted to a monitor 150,and/or stored in a storage device 135.

In addition to selecting one or more media input items to be included inthe media output item, the media studio 105 processor 110 may beprogrammed to perform various media processing operations, e.g.,scaling, mixing, morphing, compositing, adding overlays, etc., based oncommands received from the operator console 120. Additionally oralternatively, the processor 110 may be programmed to perform objecttracking, image stabilization, etc.

The network 140 represents one or more mechanisms for delivering mediacontent between the media studio 105 and the media devices 115.Accordingly, the network 140 may be one or more of various wired orwireless communication mechanisms, including any desired combination ofwired, e.g., cable, fiber, etc., and/or wireless (e.g., cellular,wireless, satellite, microwave, and radio frequency) communicationmechanisms and any desired network topology (or topologies when multiplecommunication mechanisms are utilized). Exemplary communication networksinclude wireless communication networks, local area networks (LAN) suchas a WiFi network or Ethernet, and/or wide area networks (WAN), such asthe Internet, etc.

In addition to the one or more networks 140, one or more wired orwireless direct connections 155 may be used to connect the media studio105 to the media devices 115. Direct connections 155 may include e.g.,Bluetooth, Universal Serial Bus (USB), high-definition multimediainterfaces (HDMI), custom serial interfaces, etc.

In particular, one or more high-definition multimedia interfaces (HDMI)may be used to transfer data between a media device 115 and the mediastudio 105, or from the media studio 105 to the monitor 150. HDMI is awell-known, proprietary audio/video interface for transferringuncompressed video data and compressed or uncompressed digital audiodata from an HDMI-compliant source device such as the media device 115to a digital media processing device such as the media studio 105 or tothe compatible computer monitor, e.g., the monitor 150.

The server 145 may be communicatively coupled to the media studio 105and the media devices 115 via the network 140. The server 145 mayinclude a communications circuit for communications via the network 140,and may further include memory and one or more processors configured toperform programs, i.e., sets of computer-executable instructions, storedin the memory. The server 145 may, e.g., receive media output items andstore the media output items for future use.

Media content, such as the media input items (streams), media outputitems (streams), and/or multiview media items, is generally deliveredvia the network 140 in a digital format, e.g., as compressed audioand/or video data, and may include media data and metadata. For example,Moving Picture Experts Group (MPEG) refers to a set of standardsgenerally promulgated by the International StandardsOrganization/International Electrical Commission MPEG. H.264 refers to astandard promulgated by the International Telecommunications Union(ITU). Accordingly, by way of example and not limitation, media contentmay be provided in a format such as the MPEG-1, MPEG-2, or theH.264/MPEG-4 Advanced Video Coding standards (AVC) (H.264 and MPEG-4 atpresent being consistent), HEVC/H.265, or according to some otherstandard or standards. For example, media content could be audio dataformatted according to standards such as MPEG-2 Audio Layer III (MP3),Advanced Audio Coding (AAC), etc. Further, the foregoing standardsgenerally provide for including metadata.

The camera 125 captures media data, i.e., visual and sound data such asphotographs and videos, and transmits the media data, e.g., via thenetwork 140, to the media studio 105. Examples of cameras 125 includeportable devices such as smartphones, tablets, laptops, digital cameras,etc., including one or more data collectors for collecting media data.The collected media data may include visual data and audio data.Examples of cameras 125 may further include security cameras, trafficcams, cameras transported by drones, etc.

The viewer 130 may be used to display media output data received fromthe media studio 105, and may include a display such as a liquid crystaldisplay (LCD) or plasma display. The media data may be received, forexample, via the network 140 or via the direct connection 155. Examplesof the viewer 130 include mobile devices such as mobile phones, tablets,and laptops and may further include devices such as digital televisions.The viewer 130 may receive, e.g., Full HD data, providing an imageresolution of 1920 by 1080 pixels. Data formats with other resolutionsmay also be used.

The storage device 135 may store media data and provide an interface toallow the media studio 105 to access the data via the network 140 or viathe direct connection 155. The media storage device may include one ormore types of data storage such as read only memory (ROM), random accessmemory (RAM), flash memory, electrically programmable memory (EPROM),electrically programmable and erasable memory (EEPROM), embeddedMultiMediaCard (eMMC), secure digital (SD) card, a hard drive, etc.Further, the media storage device 135 may include a processor,programmed, for example, to receive commands from the media studio 105.The processor may be further programmed, based on the commands, toretrieve media data items from data storage, and send the media dataitems to the media studio 105.

Communications between the storage device 135 and the media studio 105may be performed via the network 140. Additionally or alternatively,communications may be performed via the direct connection 155. Forexample, the storage device 135 may be connected to the media studio 105via a Universal Serial Bus (USB) port, or other wired or wirelessinterface.

The operator console 120 may be used to control the operation of themedia studio 105. The operator console 120 may include a memory and aprocessor, the memory storing program code, i.e., computer-executableinstructions, executable by the processor. The operator console 120 userinterface may be communicatively coupled to the operator console 120processor and may include one or more input devices such as amicrophone, buttons, a touchscreen display, a mouse, a keyboard, agesture-recognition device, switches, etc., for receiving input from theuser. The user interface may further include one or more output devicessuch as a display, lamps, speakers, etc. for communicating informationto the user. All or a portion of the user interface may be physicallyseparate from the operator console. For example, the operator console120 may be a tablet computer which projects its output to anotherscreen, (e.g., air-play), while the operator continues to control themedia studio 105 from the tablet computer.

The communications circuit may include a radio frequency (RF)transceiver for WiFi communications (typically 2.4 GHz or 5 GHz bands).The RF transceiver may communicate, for example, directly with a RFreceiver included in the media studio 105. Additionally oralternatively, the communications circuit may include, e.g., an Ethernetinterface, a Universal Serial Bus (USB) interface, a Bluetoothtransceiver, a high-definition multimedia interface (HDMI), etc.

In addition to commands related to selecting media input items fordisplay in the media output item, commands from the operator console 120may include instructions to perform functions such as scaling, mixing,morphing, compositing, adding overlays, etc. Further, commands from theoperator console 120 may include instructions to perform functions suchas object tracking, image stabilization, etc.

The media studio 105 may include an internal memory and a processor 110,the memory storing program code, i.e., computer-executable instructions,executable by the processor 110. The processor 110 is communicativelycoupled to the media devices 115, e.g., via the network 140. Theinternal memory may be, e.g., read only memory (ROM), random accessmemory (RAM), flash memory, electrically programmable memory (EPROM),electrically programmable and erasable memory (EEPROM), embeddedMultiMediaCard (eMMC), a hard drive, etc., and may be used to storeprograms executable by the processor 110, as well as to store, forexample, data representing inputs from the user, instructions receivedfrom the operator console 120, media data received from a remote mediadevice 115, etc.

Additionally, the media studio 105 may include a high-definition mediainterface (HDMI) for connecting to a media device 115, such as a camera125.

The media studio 105 processor 110 may, in some cases, generate amultiview media item. The multiview media item may include, for example,a picture-in-picture (PIP) display, wherein two or more of the mediainput items or streams are displayed simultaneously, e.g., side-by-side,within the multiview media item. The media studio 105 may transmit themultiview media item via the network 140 to one or more of the mediadevices 115, such as the operator console 120 or the storage devices135.

The media studio 105 processor 110 is further programmed to receivecommands from the operator console 120. Based on the commands, the mediastudio 105 may generate a media output item or stream. The media studio105 may select data from one or more of the media input items to includein the media output item. The media studio 105 may further, based oncommands received from the operator console 120, perform various mediaprocessing functions such as scaling, mixing, morphing compositing,adding overlays, tracking of specific people or objects, smart taggingetc. related to the generation of the media output item.

In addition, the media studio 105 may perform media processing functionsbased on predetermined rules for generating the media output item.Examples of rules are discussed below.

The media studio 105 processor 110 may output the media output iteme.g., to viewers 130, to the operator console 120, and/or to otherdisplay devices. Additionally or alternatively, the media studio 105 mayoutput the media output item, e.g., to a server 145, or to storagedevices 135, where the media output item may be stored for future use.

The media devices 115 such as cameras 125 may collect and transmit themedia data, e.g., via the network 140, or via the direct connection 155,to the media studio 105. The collected media data may be, e.g., fullhigh-definition (Full HD) data, providing an image resolution of 1920 by1080 pixels.

Typically, the media data is transmitted by the media device 115, e.g.,the camera 125, based on a transmission rate. Each of the transmittedmedia units (e.g., frames) may further include at least one of video andaudio data. The transmission rate defines a rate of streaming media dataincluding media units from a transmitter, e.g., the camera 125, to areceiver, e.g., the media studio 105. The transmission rate may beidentified in various units such as frames per second (fps). In oneexample, the camera 125 may stream media data with a rate of 30 fps.Thus, the camera 125 may transmit an image frame (or a media unit) each33.33333 milliseconds.

To transmit the media data from the media device 115 at a given rate,e.g., 30 fps, the media device 115 may include an electronic circuitsuch as a processor to capture and transmit the media data. In oneexample, the processor is programmed to capture an image frame andtransmit the captured image frame based on a running clock of the mediadevice 115. For example, the media device 115 may include an electronicoscillator circuit that uses a mechanical resonance of a vibratingcrystal of piezoelectric material to create an electrical signal with aprecise frequency. The media device 115 may operate based on a clockrate of e.g., 1 kilohertz (kHz) based on the output of the electronicoscillator circuit. The media device 115 processor may count the clockpulses generated by the running clock each 1 ms to determine a time ofcapturing and transmitting a new image frame. In this example, startingfrom a time 0 (zero), the processor of the media device 115 may captureimage frames at 0, 33 ms, 66 ms, etc., because based on the clock rateof 1 kHz, the media device 115 is unable to generate timestamps having amore accurate resolution than one millisecond, e.g., it cannot generatetimestamps of 33.33333 ms, 66.66666 ms, etc. Additionally oralternatively, the clock rate of the media device 115 may be inaccurate.For example, a time difference between two consecutive clock pulses maybe different from 1 ms, e.g., 1.1 ms. That inaccuracy may then cause atransmission of a frame at a time index of 33.1 ms instead of at anexpected time index of 33.33.

A time of capturing and/or transmitting the media unit may be recordedin a so called time stamp associated with the transmitted media unit.For example, the transmitted media unit may include a time stamp datarecord that includes the associated time. Thus, a receiver of the mediaunit may identify the time associated with the media unit by thetransmitter based on the included time stamp. Various media streamingtechnologies provide different mechanisms for associating a time stampto the transmitted media data.

Further, a receiver of the streamed media data, e.g., the media studio105, may have a different clock rate than a media capture device 115.For example, the media studio 105 may have a clock rate of 1000 kHz,i.e., a clock pulse duration of 1 microsecond (μs). Thus, with referenceto the above example of 30 fps, the media studio 105 processor 110 maybe programmed to expect the media units at 0, 33.333ms, 66.666 ms, etc.Therefore, based on differences in clock rates of the transmitter versusthe receiver, and/or an inaccuracy of a media transmitting device 115clock rate, a mismatch may be caused between the time stamp associatedwith the media unit and a time stamp that the receiver expects, e.g., 33ms versus 33.333 ms. This mismatch may impair one or more operations ofthe media studio 105. In one example, the media studio 105 processor 110may be programmed to generate a multiview media item including mediastreams respectively received from two cameras 125. The mismatch of timestamps associated with the received media units from the cameras 125 andthe media studio 105 may impair media studio 105 in generating amultiview item, or may prevent such operation altogether.

With reference to FIG. 2, the media studio 105 processor 110 may beprogrammed to receive first and second media units F₀, F₁ withrespective first and second time stamps t₀, t₁ that are assigned basedon a first clock cycle time C₁ and a data transmission rate R. Theprocessor 110 may be further programmed to assign an adjusted time stampt₁+Δ₁ to the second media unit F₁ based on the first clock cycle timeC₁, a second clock cycle time C₂, the first time stamp t₀, and the datatransmission rate R. In this example, t₁ is a timestamp of the unit F₁when it is received by the studio 105, and is Δ₁ is an amount of time bywhich the time t₁ is adjusted, i.e., in this example, an amount of timeadded to the time t₁.

With reference to the example described above, the camera 125 maytransmit the media units F₀, F₁ at t₀, t₁ corresponding to 0 ms and 33ms. The processor 110 may be programmed to assign the adjusted timestamps 0 ms and 33.333 ms based on the first clock cycle time C₁ of 1ms, the second clock cycle time C₂ of 1 μs, the second time stamp 33 ms,and the transmission rate of 30 fps. Thus, the processor 110 maydetermine that Δ₁ is 0.333 ms. As another example, with a second cycletime C₂ of 10 megaherz (MHz), the processor 110 may be programmed todetermine that Δ₁ is 3.3 μs.

In general, the adjustment value A represents a time adjustmentassociated with a time stamp and can be either a positive or a negativevalue. For example, a third media unit F₂ may have a time stamp of 67ms. The processor 110 may be programmed to assign an adjusted third timestamp 66.666 ms based on the first clock cycle time C₁ of 1 ms, thesecond clock cycle time C₂ of 1 μs, the third time stamp 67 ms, and thetransmission rate of 30 fps. Thus, the processor 110 may determine thatΔ₁ is −0.334 ms. As another example, with a second cycle time C₂ of 10megahertz (MHz), the processor 110 may be programmed to determine a Δ₁of −3.4 μs.

In one example, an adjusted time stamp may be a nearest possible timestamp to the received media unit time stamp that is divisible by thereceiver clock cycle time. With reference to the example above, the timestamp of the received frame F₂ is 67 ms. Based on the transmission rateR and the receiver clock cycle time C₂, an expected time τ between twoconsecutive frames is 33.333 ms. Here should be noted that 33.333 ms isdivisible by the clock cycle time C₂ of 1 μs, whereas, e.g., 33.3333 ms,is not divisible by the 1 ms. Thus, possible adjusted time stamps mayinclude 66.666 ms, 99.999 ms, etc. Therefore, nearest possible stamp tothe received media unit time stamp of 67 ms is 66.666 ms. That resultsin the adjustment value Δ₁ of −0.334 ms.

As shown in the examples above, the second clock cycle time C₂ of 1 μsis less than the first clock cycle time C₁ of 1 ms. Alternatively, thesecond clock cycle time C₂ may be greater than the first clock cycletime C₁. For example, the processor 110 may be programmed to assign anadjusted time stamp of 33 ms to a media unit received with a time stampof 33.333 ms, based on the transmission rate of 30 fps, a first clockcycle time C₁ of 1 μs, and a second clock cycle time of 1 ms.

As another example, the processor 110 may be programmed to replace thesecond time stamp t₁ with the adjusted time stamp t₁+Δ₁. Additionally oralternatively, the processor 110 may be programmed to associate aseparate data record to the second media unit F₁ that holds the adjustedtime stamp t₁+Δ₁.

With reference to FIG. 3, the processor 110 may be programmed to assignan adjusted time stamp to a media unit only upon determining that thesecond time stamp is within an expected time interval. For example, theprocessor 110 may be programmed to discard a third media unit F₂, upondetermining that the third time stamp t₂ is outside an expected timeinterval W₁. The expected time interval W₁ may be defined based on anexpected time stamp and a maximum deviation threshold. For example, anexpected time interval Wi for an expected time stamp of 66.666 ms and amaximum deviation threshold of 750 μs may include any time between65.916 ms and 67.416 ms.

In one example, the processor 110 may be programmed to determine themaximum deviation threshold based on the first clock cycle time C₁ , andthe second clock cycle time C₂. Thus, the expected time interval W₁ maybe in part based on the transmission rate R, the first clock cycle timeC₁, and the second clock cycle time C₂. For example, the processor 110may be programmed to determine the maximum deviation threshold bydetermining maximum of the clock cycle times C₁, C₂. With reference tothe above example, the maximum deviation threshold may be 1 ms which isthe maximum of 1 ms and 1 μs. As another example, the processor 110 maybe programmed to determine the expected time interval W₁ based on thefirst clock cycle time C₁ and a plurality of time stamps t₀, t₁associated with a plurality of last received media units F₀, F₁. Forexample, the processor 110 may be programmed to determine an inaccuracyof the first clock cycle time C₁ of the transmitter based on a pluralityof consecutive media units F₀, F₁. The processor 110 may be furtherprogrammed to determine the expected time interval W₁ at least in partbased on the determined inaccuracy of the transmitter. For example, theprocessor 110 may be programmed to determine an average inaccuracy oftime stamps of last 10 media units by comparing timestamps to respectiveexpected receiving times and multiple the average inaccuracy by 3. Thus,if the average inaccuracy of time stamps of the last 10 media units is 1ms, the processor 110 may determine that the maximum deviation thresholdis 3 ms which is 1 ms multiplied by factor 3.

In one example, the processor 110 may be further programmed to re-usethe media unit F₁ as a next unit in a media stream instead of the mediaunit F₂, upon discarding the second media unit. Thus, advantageously,the receiver, e.g., the media studio 105, is not provided a media streamwith a missing media unit, i.e., no until at a time index when the mediaunit F₂ is expected. For example, as shown in FIG. 3, the processor 110may be programmed to re-use the media unit F₁ instead of expected mediaunit F₂ by assigning an adjusted time stamp t₁Δ₁+τ. Thus, the processor110 may be programmed to reuse (or reinstitute) the media unit F₁ for asecond time with a time stamp t₁+Δ₁+τ. Thus, the re-used media unit F1with the time stamp t₁+Δ₁τ may be a replacement for the discarded secondmedia unit.

Alternatively, as shown in FIG. 4A, the processor 110 may be programmedto prevent an adjustment of the second time stamp t₂ upon determiningthat the second time stamp t₂ is outside an expected time interval W₁.In other words, the processor 110 may be programmed to process, e.g.,reproduce, the media unit F₂ using the time stamp t₂ without anyadjustment thereof.

The processor 110 may be programmed to receive, subsequent to the mediaunit F₂, a third media unit F₃ with a time stamp t₃, and assign anadjusted third time stamp t₃+Δ₃ to the third media input F₃ based atleast in part on the second time stamp t₂. In one example, as shown inTable 1, the processor 110 may be programmed to expect the third mediainput F₃ in a time that is based in part on the time stamp of the mediainput F₂. In this example, the processor 110 may be programmed to expectthe third media input F₃ at 108.333 ms which is 33.333 ms after thereceived time stamp t₂, rather than at 99.999 ms which is 33.333 msafter an expected second time stamp for the second media input F₂.

The processor 110 may be programmed to determine a second expected timeinterval W₂ for the third media input F₃, based at least in part on thesecond time stamp t₂. For example, with a given maximum deviationthreshold of 750 μs (as described with reference to FIG. 3), theprocessor 110 may be programmed to determine the second time interval W₂to be between 107.583 ms and 109.083 ms, i.e., a time between 108.333ms-750 μs and 108.333 ms+750 μs.

TABLE 1 Expected Received Adjusted timestamp (ms) timestamp (ms)timestamp (ms) 0 0 0 33.333 33.000 33.333 66.666 75.000 75.000 108.333108.000 108.333

FIG. 4B is a graph showing a last media unit repeated until a next mediaunit within an expected time interval is received. The processor 110 maybe programmed to re-use, e.g., repeat, the last received media unit F₁upon determining that a second time stamp t₂ of a next frame immediatelyfollowing the media unit F₁ is outside a second time threshold, e.g.,2τ, which is greater than the expected time interval W₁. Re-using orrepeating the last media unit means reproducing (or rerecording) samecontent, e.g., audio and/or video data, of the last media unit, e.g.,the media unit F₁, with new time stamps t₁+τ, t₁+3τ, t₁+3τetc. A numberof repetitions n of the last media unit F₁ may be determined based onthe operation (1).

$\begin{matrix}{n = {\left( \frac{{t\; 2} - {t\; 1}}{\tau} \right) - 1}} & (1)\end{matrix}$

PROCESSING

FIG. 5 is a flowchart of an exemplary process 500 for synchronization ofmedia units received at a receiver such as the media studio 105. Forexample, the media studio 105 processor 110 may be programmed to executeblocks of the process 500.

The process 500 begins in a block 505, in which the processor 110receives system 100 information such as transmission rates, clock cycletimes, etc., of media devices 115 and the media studio 105. In oneexample, the processor 110 may be programmed to receive informationassociated with the media devices 115 such as the transmission rate R orthe clock cycle times C₁, C₂ by sending a request via the network 140 tothe respective media device 115, and then receiving a reply from therespective media device 115. Additionally or alternatively, theprocessor 110 may be programmed to determine the system 100 information,e.g., based on stored information in the media studio 105 memory.

Next, in a block 510, the processor 110 determines expected timeintervals W for the media units received from a transmitter. Forexample, the processor 110 may determine the expected time interval Wfor media units of the camera 125 based on the processor 110 clock cycletime C₂, the camera 125 transmission rate R, the camera 125 clock cycletime C₁, and the time stamp of a last received media unit. Note that theprocessor 110 may be programmed to determine the expected time intervalW after at least receiving two media units; as described above, theexpected time interval W may be defined based on a time stamp of thelast received media unit.

Next, in a block 515, the processor 110 receives a media unit, e.g., animage frame. The received media unit may include a time stamp that isdetermined by the transmitter of the media unit. Alternatively, theprocessor 110 may be programmed to determine the time stamp of areceived media unit based on a time of receipt of the media unit. Forexample, the processor 110 may be programmed to associate a time stamp33.11 ms when the transmitter captures and transmits the media unit at33 ms, and a travel time from the transmitter to the receiver is 110 μs.In this example, the processor 110 may be programmed to use this-assigned-by-receiver time stamp as the time stamp of the received mediaunit which may be further adjusted by the processor 110, as discussedbelow.

Next, in a decision block 520, the processor 110 determines whether thereceived media unit is within the expected time interval. For example,the processor 110 may have determined the expected time interval to bebetween 33 ms and 34 ms. Then, the received media unit F₁ with the timestamp t₁ may be within the expected time interval of 33-34 ms. Asanother example, as shown in FIG. 3, the received media unit F₂ may havea time stamp t₂ that is outside the expected time interval W₁. If theprocessor 110 determines that the time stamp of the received media unitis within the expected time interval, then the process 500 proceeds to ablock 525; otherwise the process 500 proceeds to a block 530.

In the block 525, the processor 110 assigns an adjusted time stamp tothe received media unit. As described above, the processor 110 may beprogrammed to determine an adjusted time stamp based on the transmissionrate R, the clock cycle time C1 of the transmitter, the clock cycle timeC2 of the receiver, e.g., the media studio 105, the time stamp of thereceived media unit.

In the block 530, the processor 110 prevents an adjustment of the timestamp of the received media input. For example, the processor 110 may beprogrammed to process, e.g., reproduce, the received media input withoutadjusting the timestamp thereof. The processor 110 may be programmed todetermine an expected time interval for a subsequent media input basedon the time stamp of the received media input. Alternatively, theprocessor may be programmed to discard the received media unit. Forexample, with reference to FIG. 3, the processor 110 may be programmedto discard the information provided in the media studio 105 memory thatincludes image and/or audio data associated with the received media unitF₂. In this example, the processor 110 may re-use last received mediaunit. For example, as shown in FIG. 3, the processor 110 may beprogrammed to re-use the media unit F₁ that was received prior to themedia unit F2.

In a decision block 540, the processor 110 determines whether mediastreaming is stopped. For example, the processor 110 may determine thatthe media stream has stopped based on receiving a message from a mediadevice 115 including a communication termination notification.Additionally or alternatively, the processor 110 may determine that themedia streaming has stopped based on determining that at least aspecific number, e.g., 10, of consecutive media units have not beenreceived. If the processor 110 determines that the media streamingstopped, then the process 500 ends; otherwise the process 500 returns tothe block 510.

FIG. 6 is a flowchart of another exemplary process 600 for adjustingtime stamps of media units received at a receiver such as the mediastudio 105. For example, the media studio 105 processor 110 may beprogrammed to execute blocks of the process 600.

The process 600 begins in a block 605, in which the processor 110receives system 100 information such as transmission rates, clock cycletimes, etc., of media devices 115 and the media studio 105. In oneexample, the processor 110 may be programmed to receive informationassociated with the media devices 115 such as the transmission rate R orthe clock cycle times C₁, C₂ by sending a request via the network 140 tothe respective media device 115, and then receiving a reply from therespective media device 115. Additionally or alternatively, theprocessor 110 may be programmed to determine the system 100 information,e.g., based on stored information in the media studio 105 memory.

Next, in a block 610, the processor 110 determines expected timeintervals W for the media units received from a transmitter. Forexample, the processor 110 may determine the expected time interval Wfor media units of the camera 125 based on the processor 110 clock cycletime C₂, the camera 125 transmission rate R, the camera 125 clock cycletime C₁, and the time stamp of a last received media unit. Note that theprocessor 110 may be programmed to determine the expected time intervalW after at least receiving two media units; as described above, theexpected time interval W may be defined based on a time stamp of thelast received media unit.

Next, in a block 615, the processor 110 receives a media unit, e.g., animage frame. The received media unit may include a time stamp that isdetermined by the transmitter of the media unit. Alternatively, theprocessor 110 may be programmed to determine the time stamp of areceived media unit based on a time of receipt of the media unit. Forexample, the processor 110 may be programmed to associate a time stampof 33.11 ms when the transmitter captures and transmits the media unitat 33 ms, and a travel time from the transmitter to the receiver is 110μs. In this example, the processor 110 may be programmed to use thisassigned-by-receiver time stamp as the time stamp of the received mediaunit which may be further adjusted by the processor 110, as discussedbelow.

Next, in a decision block 620, the processor 110 determines whether thereceived media unit is within the expected time interval, e.g., the timeinterval W₁. For example, the processor 110 may have determined theexpected time interval to be between 33 ms and 34 ms. Then, the receivedmedia unit F₁ with the time stamp t₁ may be within the expected timeinterval of 33-34 ms. As another example, as shown in FIG. 3, thereceived media unit F₂ may have a time stamp t₂ that is outside theexpected time interval W₁. In yet another example, as shown in FIG. 4B,the received media unit F₂ may have a time stamp t₂ that is outside theexpected time interval W₁ and beyond the second time threshold, e.g.,2τ. If the processor 110 determines that the time stamp of the receivedmedia unit is within the expected time interval, then the process 600proceeds to a block 625; otherwise the process 600 proceeds to adecision block 630.

In the block 625, the processor 110 assigns an adjusted time stamp tothe received media unit. As described above, the processor 110 may beprogrammed to determine an adjusted time stamp based on the transmissionrate R, the clock cycle time C1 of the transmitter, the clock cycle timeC2 of the receiver, e.g., the media studio 105, the time stamp of thereceived media unit. Following the block 625, the process 600 proceedsto a decision block 645.

In the decision block 630, the processor 110 determines whether a mediaunit is received within the second time threshold. The processor 110 maybe programmed to determine whether a media unit is received within thesecond time threshold, e.g., 2τ(i.e., if t₂-t₁<2τ). If the processor 110determines that a media unit is received within the second timethreshold, then the process 600 proceeds to a block 640; otherwise theprocess 600 proceeds to a block 635.

In the block 635, the processor 110 repeats the last media unit. Forexample, as shown in FIG. 4B, the processor 110 may be programmed torepeat the last media unit F₁ with an expected time stamp, e.g., at atime t₁+τ. The expected time stamp is determined based on the lastreproduced media unit, the transmission rate R, etc. Thus, if the lastmedia unit is already repeated at the time t₁+τ, the processor 110 maydetermine the expected time t₁+2τ. In other words, the processor 110 mayrepeat the frame F₁ at the time t₁ +Zr. Following the block 635, theprocess 600 returns to the decision block 620

In the block 640, the processor 110 discards the received media unit andrepeats the last media unit. For example, with reference to FIG. 3, theprocessor 110 may be programmed to discard the information provided inthe media studio 105 memory that includes image and/or audio dataassociated with the received media unit F₂, and to repeat the lastreceived media unit. For example, as shown in FIG. 3, the processor 110may be programmed to re-use the media unit F₁ that was received prior tothe media unit F₂. Following the block 640, the process 600 proceeds tothe decision block 645.

In a decision block 645, the processor 110 determines whether mediastreaming is stopped. For example, the processor 110 may determine thatthe media stream has stopped based on receiving a message from a mediadevice 115 including a communication termination notification.Additionally or alternatively, the processor 110 may determine that themedia streaming has stopped based on determining that at least aspecific number, e.g., 10, of consecutive media units have not beenreceived. If the processor 110 determines that the media streamingstopped, then the process 600 ends; otherwise the process 600 returns tothe block 610.

Advantageously, when timestamps of a media stream are adjusted based onthe present disclosure, a table size of media stream, e.g., an MP4format video and audio stream, may be reduced compared to when the timestamps are not adjusted. In one example, sizes of known tables“Sample-to-Time” and “Sync Sample Atoms” for storing the information ofaudio and video data may be reduced.

The article “a” modifying a noun should be understood as meaning one ormore unless stated otherwise, or context requires otherwise. The phrase“based on” encompasses being partly or entirely based on.

Computing devices as discussed herein generally each includeinstructions executable by one or more computing devices such as thoseidentified above, and for carrying out blocks or steps of processesdescribed above. Computer-executable instructions may be compiled orinterpreted from computer programs created using a variety ofprogramming languages and/or technologies, including, withoutlimitation, and either alone or in combination, Java™, C, C++, VisualBasic, Java Script, Perl, HTML, etc. In general, a processor (e.g., amicroprocessor) receives instructions, e.g., from a memory, acomputer-readable medium, etc., and executes these instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein. Such instructions and other data may be stored andtransmitted using a variety of computer-readable media. A file in thecomputing device is generally a collection of data stored on a computerreadable medium, such as a storage medium, a random access memory, etc.

A computer-readable medium includes any medium that participates inproviding data (e.g., instructions), which may be read by a computer.Such a medium may take many forms, including, but not limited t₀,non-volatile media, volatile media, etc. Non-volatile media include, forexample, optical or magnetic disks and other persistent memory. Volatilemedia include dynamic random access memory (DRAM), which typicallyconstitutes a main memory. Common forms of computer-readable mediainclude, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, any other magnetic medium, a CD-ROM, DVD, any otheroptical medium, punch cards, paper tape, any other physical medium withpatterns of holes, a RAM, a PROM, an EPROM, a FLASH, an EEPROM, anyother memory chip or cartridge, or any other medium from which acomputer can read.

With regard to the media, processes, systems, methods, etc. describedherein, it should be understood that, although the steps of suchprocesses, etc. have been described as occurring according to a certainordered sequence, such processes could be practiced with the describedsteps performed in an order other than the order described herein. Itfurther should be understood that certain steps could be performedsimultaneously, that other steps could be added, or that certain stepsdescribed herein could be omitted. In other words, the descriptions ofsystems and/or processes herein are provided for the purpose ofillustrating certain embodiments, and should in no way be construed soas to limit the disclosed subject matter.

Accordingly, it is to be understood that the present disclosure,including the above description and the accompanying figures and belowclaims, is intended to be illustrative and not restrictive. Manyembodiments and applications other than the examples provided would beapparent to those of skill in the art upon reading the abovedescription. The scope of the invention should be determined, not withreference to the above description, but should instead be determinedwith reference to claims appended hereto and/or included in anon-provisional patent application based hereon, along with the fullscope of equivalents to which such claims are entitled. It isanticipated and intended that future developments will occur in the artsdiscussed herein, and that the disclosed systems and methods will beincorporated into such future embodiments. In sum, it should beunderstood that the disclosed subject matter is capable of modificationand variation.

What is claimed is:
 1. A system, comprising a processor and a memory,the memory storing instructions executable by the processor to: receivefirst and second media units with respective first and second timestamps that are assigned based on a first clock cycle time and a datatransmission rate; and assign an adjusted time stamp to the second mediaunit based on the first clock cycle time, a second clock cycle time, thefirst time stamp, and the data transmission rate.
 2. The system of claim1, wherein the adjusted time stamp is a nearest time stamp to the secondtime stamp based on the transmission rate, and is divisible by thesecond clock cycle time.
 3. The system of claim 1, wherein the secondclock cycle time is less than the first clock cycle time.
 4. The systemof claim 1, wherein the instructions include further instructionsexecutable by the processor to assign the adjusted time stamp to thesecond media unit only upon determining that the second time stamp iswithin an expected time interval.
 5. The system of claim 4, wherein theinstructions include further instructions to discard the second mediaunit upon determining that the second time stamp is outside the expectedtime interval.
 6. The system of claim 5, wherein the instructionsinclude further instructions to discard the second media unit only upondetermining that the second time stamp is outside the expected timeinterval and within a second time threshold greater than the expectedtime interval.
 7. The system of claim 6, wherein the instructionsinclude further instructions to re-use the first media unit instead ofthe second media unit upon discarding the second media unit.
 8. Thesystem of claim 7, wherein the instructions include further instructionsto re-use the first media unit by assigning the adjusted time stamp tothe first media unit and processing the first media unit with theadjusted time stamp as a replacement for the discarded second mediaunit.
 9. The system of claim 4, wherein the instructions include furtherinstructions to determine the expected time interval based on the firstclock cycle time and a plurality of time stamps associated with aplurality of last received media units.
 10. The system of claim 4,wherein the instructions include further instructions to determine theexpected time interval based on an expected time stamp and a maximumdeviation threshold.
 11. The system of claim 10, wherein theinstructions include further instructions to prevent an adjustment ofthe second time stamp upon determining that the second time stamp isoutside the expected time interval.
 12. The system of claim 10, whereinthe instructions include further instructions to determine a secondexpected time interval for a third media unit that is expectedsubsequent to the second media unit, based at least in part on thesecond time stamp.
 13. The system of claim 10, wherein the instructionsinclude further instructions to: receive a third media unit, subsequentto the second media unit, with a third time stamp; and assign anadjusted third time stamp to the third media unit based at least in parton the second time stamp.
 14. The system of claim 1, wherein theinstructions include further instructions to repeat the first media unitupon determining that the second time stamp is outside an expected timeinterval and outside a second time threshold greater than the expectedtime interval.
 15. A method comprising: receiving first and second mediaunits with respective first and second time stamps that are assignedbased on a first clock cycle time and a data transmission rate; andassigning an adjusted time stamp to the second media unit based on thefirst clock cycle time, a second clock cycle time, the first time stamp,and the data transmission rate.
 16. The method of claim 15, furthercomprising discarding the second media unit only upon determining thatthe second time stamp is outside an expected time interval and within asecond time threshold greater than the expected time interval.
 17. Themethod of claim 15, further comprising repeating the first media unitupon determining that the second time stamp is outside an expected timeinterval and outside a second time threshold greater than the expectedtime interval.
 18. A system, comprising: a client device computerprogrammed to transmit first and second media units; and a media studiocomputer programmed to: receive, from a media device, first and secondmedia units with respective first and second time stamps that areassigned based on a first clock cycle time and a data transmission rate;and assign an adjusted time stamp to the second media unit based on thefirst clock cycle time, a second clock cycle time, the first time stamp,and the data transmission rate.
 19. The system of claim 18, wherein themedia studio computer is further programmed to repeat the first mediaunit upon determining that the second time stamp is outside an expectedtime interval and outside a second time threshold greater than theexpected time interval.
 20. The system of claim 18, wherein the mediastudio computer is further programmed to discard the second media unitonly upon determining that the second time stamp is outside an expectedtime interval and within a second time threshold greater than theexpected time interval.